# Mixture of Experts

DeepSeek-VL2 is a series of advanced large-scale Mixture of Experts (MoE) visual language models, significantly improved compared to its predecessor, DeepSeek-VL. This model series demonstrates exceptional capabilities across various tasks, including visual question answering, optical character recognition, document/table/chart understanding, and visual localization. The DeepSeek-VL2 series consists of three variants: DeepSeek-VL2-Tiny, DeepSeek-VL2-Small, and DeepSeek-VL2, with 1.0B, 2.8B, and 4.5B active parameters, respectively. DeepSeek-VL2 achieves competitive or state-of-the-art performance compared to existing open-source dense and MoE-based models with similar or fewer active parameters.

DeepSeek-V2.5-1210 is an upgraded version of DeepSeek-V2.5, improving multiple capabilities including mathematics, coding, and writing inference. The model's performance in the MATH-500 benchmark has increased from 74.8% to 82.8%, while its accuracy in the LiveCodebench (08.01 - 12.01) benchmark has risen from 29.2% to 34.38%. Additionally, the new version enhances user experience in file uploads and web summarization features. The DeepSeek-V2 series (including basics and chat capabilities) supports commercial use.

Yuan2.0-M32 is a mixture of experts (MoE) language model featuring 32 experts, of which 2 are active. It introduces a new routing network—an attention router—to improve the efficiency of expert selection, resulting in a 3.8% accuracy boost over models using traditional routing networks. Yuan2.0-M32 was trained from scratch using 200 billion tokens, with a computational cost only 9.25% of that required by similarly parameterized dense models. It demonstrates competitive performance in coding, mathematics, and various professional fields, with only 370 million active parameters out of a total of 4 billion, and a forward computation requirement of just 7.4 GFLOPS per token, which is only 1/19th of Llama3-70B's requirements. In MATH and ARC-Challenge benchmark tests, Yuan2.0-M32 outperformed Llama3-70B, achieving accuracies of 55.9% and 95.8%, respectively.